encoding/json: Add Custom JSON Package with Build Tag Support for Sonic #1623

shazbert · 2024-08-20T03:07:12Z

PR Description

This pull request introduces a custom JSON package located in encoding/json/ to facilitate swapping between the default Go encoding/json package and the sonic library based on build tags.

Adds:

json.go (default implementation using encoding/json).
sonic.go (alternative implementation using sonic library).

Import Updates:

All imports to the standard encoding/json package have been updated to the new custom package github.com/thrasher-corp/gocryptotrader/encoding/json.

Added basic benchmarks for both the default encoding/json and the sonic implementation to evaluate hot-swap performance.

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and
also consider improving test coverage whilst working on a certain feature or package.

go test ./... -race
golangci-lint run
Test X

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation and regenerated documentation via the documentation tool
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally and on Github Actions with my changes
Any dependent changes have been merged and published in downstream modules

codecov · 2024-08-20T03:13:38Z

Codecov Report

Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 37.07%. Comparing base (72a2a16) to head (135e2b6).

Files with missing lines	Patch %	Lines
cmd/exchange_wrapper_issues/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1623   +/-   ##
=======================================
  Coverage   37.07%   37.07%           
=======================================
  Files         414      414           
  Lines      180274   180274           
=======================================
+ Hits        66835    66845   +10     
- Misses     105574   105577    +3     
+ Partials     7865     7852   -13

Files with missing lines	Coverage Δ
backtester/btcli/main.go	`0.00% <ø> (ø)`
backtester/config/backtesterconfig.go	`88.46% <ø> (ø)`
backtester/config/strategyconfig.go	`83.51% <ø> (ø)`
backtester/config/strategyconfigbuilder/main.go	`0.00% <ø> (ø)`
backtester/eventhandlers/statistics/statistics.go	`74.63% <ø> (ø)`
backtester/main.go	`0.00% <ø> (ø)`
cmd/apichecker/apicheck.go	`38.20% <ø> (ø)`
cmd/config/config.go	`8.62% <ø> (ø)`
cmd/config_builder/builder.go	`0.00% <ø> (ø)`
cmd/documentation/documentation.go	`0.00% <ø> (ø)`
... and 89 more

... and 11 files with indirect coverage changes

shazbert · 2024-10-10T19:15:44Z

Flagged this as medium because this is beast. Sorry for those that cant use it. 🙏

gloriousCode

Nice work on finding this. I like the way you've implemented it and makes it easy to experiment with out parsers in future with your groundwork

orig:

// BenchmarkUnmarshal-10    	  949106	      1065 ns/op	     816 B/op	      24 allocs/op

sonic:

// BenchmarkUnmarshal-10    	 1818868	       643.8 ns/op	     951 B/op	      19 allocs/op

encoding/json/sonic_benchmark_test.go

gloriousCode · 2024-08-20T06:56:48Z

.github/workflows/tests.yml

          - os: windows-latest
            goarch: amd64
            psql: true
            skip_wrapper_tests: true
+            sonic: false
+          - os: ubuntu-latest-sonic


I think you should stick to the same style as the existing builds names with the false, false,false,true, would also just be fine with including sonic somewhere in the build name. Maybe its just not showing up below because its not merged?

Also, do we need two of each build? Would one extra one with sonic enabled suffice?

I think you should stick to the same style as the existing builds names with the false, false,false,true, would also just be fine with including sonic somewhere in the build name. Maybe its just not showing up below because its not merged?

That should have been reverted. I think.

Also, do we need two of each build? Would one extra one with sonic enabled suffice?

@thrasher- Confirming entire suite test or can just have one?

Can target amd64 linux for now since that's our main deployment OS in use ATM

encoding/json/sonic.go

encoding/json/sonic_benchmark_test.go

gloriousCode · 2025-01-09T23:09:41Z

.github/workflows/tests.yml

@@ -13,22 +13,52 @@ jobs:
            goarch: amd64


Unrelated to this line, would you consider a log on GCT start to say which one is enabled? Yes, you have build tags, but then it puts it in the logs of the application

Could add to the build tag files to call like json.Version() where its fmt.Println("hello my name is Sonic and I gotta go fast, okay?") as an example

needs checking

gloriousCode

My comments feel very petty, but I'm happy overall with the work you've done

.golangci.yml

gloriousCode · 2025-01-27T23:28:55Z

README.md

@@ -77,6 +77,7 @@ However, we welcome pull requests for any exchange which does not match this cri
 + WebGUI (discontinued).
 + Exchange HTTP mock testing. See [mock](/exchanges/mock/README.md).
 + Exchange multichain deposits and withdrawals for specific exchanges. See [multichain transfer support](/docs/MULTICHAIN_TRANSFER_SUPPORT.md).
+ **Sonic JSON Integration**: Swap between the default Go 'encoding/json' package and the sonic library using go build tags for optimized JSON handling `-tags=sonic` or by using the make file `make sonic`


Please keep styling in line with the others (or update the others) and update the template as well.

gloriousCode · 2025-01-27T23:37:11Z

encoding/json/json.go

+
+package json
+
+import "encoding/json" //nolint:depguard // This is a wrapper package


Please update comment. This PR has no readme of new functionality being introduced. Explaining that its here for the rest of the application to use the default go json package would be nice
edit: finding it hard to use my own words. But saying that its the default json package used for GCT and that all uses of json should refer to this package would be good

gloriousCode · 2025-01-27T23:38:01Z

encoding/json/common.go

@@ -0,0 +1,13 @@
+package json
+
+import "encoding/json" //nolint:depguard // This is a wrapper package


Update comment, but less detail needed than the other one

gloriousCode · 2025-01-27T23:40:35Z

encoding/json/sonic.go

+func init() {
+	log.Printf("Using %s for JSON encoding\n", Version)
+}


Makes more sense to have this in common.go because it doesn't output the default package
Also would recommend it being a function to call at a later stage. Only because its ugly! 😆 I don't mind if its not done, but it gets in front of our sweet banner

2025/01/28 10:40:52 Using encoding/json for JSON encoding ______ ______ __ ______ __ / ____/____ / ____/_____ __ __ ____ / /_ ____ /_ __/_____ ______ ____/ /___ _____ / / __ / __ \ / / / ___// / / // __ \ / __// __ \ / / / ___// __ // __ // _ \ / ___/ / /_/ // /_/ // /___ / / / /_/ // /_/ // /_ / /_/ // / / / / /_/ // /_/ // __// / \____/ \____/ \____//_/ \__, // .___/ \__/ \____//_/ /_/ \__,_/ \__,_/ \___//_/ /____//_/ GoCryptoTrader v0.1 arm64 go1.23.4 pre-release. This version is pre-release and is not intended to be used as a production ready trading framework or bot - use at your own risk. Copyright (c) 2014-2025 The GoCryptoTrader Developers. GitHub: https://github.com/thrasher-corp/gocryptotrader Kanban: https://github.com/orgs/thrasher-corp/projects/3 Slack: https://join.slack.com/t/gocryptotrader/shared_invite/enQtNTQ5NDAxMjA2Mjc5LTc5ZDE1ZTNiOGM3ZGMyMmY1NTAxYWZhODE0MWM5N2JlZDk1NDU0YTViYzk4NTk3OTRiMDQzNGQ1YTc4YmRlMTk Issues: https://github.com/thrasher-corp/gocryptotrader/issues 2025/01/28 10:40:52 Loading config file /Users/scottg/.gocryptotrader/config.json.. exchange CoinbasePro authenticated API support disabled due to default/empty APIKey/Secret/ClientID values Logger initialised.

gloriousCode · 2025-01-27T23:50:04Z

.github/workflows/tests.yml

          - os: windows-latest
            goarch: amd64
            psql: true
            skip_wrapper_tests: true
+            sonic: false
+          - os: ubuntu-latest


I'm not seeing which one is sonic in anyway from the build list.

Or is there two that use sonic (the last two ubuntu builds)? Am I blind? 🙈

The last one is a docker build so it only comes up with false true, but the matrix build above, seems to be the only one with sonic set to true, but in the logs, it doesn't seem to pull the bytedance package down. I am trying to fix that now.

Unintended catch by you sir 🚀 but that should be good to go now.

Looks like the docker build is ded, but otherwise, looks good

Ryan O'Hara-Reid added 3 commits August 20, 2024 11:37

tag optional sonic and allow full library conversion

8333e5c

Add workflow and disallow arm and darwin usage

c47137c

Add basic hotswap benchmark

6e24f60

shazbert added the blocked label Aug 20, 2024

shazbert self-assigned this Aug 20, 2024

Ryan O'Hara-Reid added 3 commits August 20, 2024 13:20

linter: fix

b70b9e7

use bash

84dd409

linter: fix?

711db77

shazbert added review me This pull request is ready for review low priority This enhancement or update will be implemented at a later date. and removed blocked labels Aug 20, 2024

Ryan O'Hara-Reid added 2 commits August 20, 2024 15:26

Fix whoopsie, add to make file, also add mention in features list.

223f786

Merge branch 'master' into encoding_swap

9a72688

shazbert added the rebase/merge of master required label Sep 2, 2024

Merge branch 'master' into encoding_swap

2f69362

shazbert removed the rebase/merge of master required label Sep 2, 2024

Ryan O'Hara-Reid added 5 commits September 13, 2024 11:22

Merge branch 'master' into encoding_swap

79c85c9

Merge branch 'master' into encoding_swap

b4f0243

Merge branch 'master' into encoding_swap

417e0aa

Merge branch 'master' into encoding_swap

83f1d57

Merge branch 'master' into encoding_swap

f2d56cf

shazbert added medium priority and removed low priority This enhancement or update will be implemented at a later date. labels Oct 10, 2024

Ryan O'Hara-Reid added 5 commits October 23, 2024 13:51

Merge branch 'master' into encoding_swap

7bd7cc4

Merge branch 'master' into encoding_swap

79ebfa1

test enforcement

6b6fb5e

actually read documentation see if this works

ac47cc6

linter: fix

135e2b6

Ryan O'Hara-Reid added 3 commits November 26, 2024 10:10

Merge branch 'master' into encoding_swap

2d08549

Merge branch 'master' into encoding_swap

1a82764

linter: fix

1ca8eaa

gloriousCode requested changes Jan 9, 2025

View reviewed changes

Ryan O'Hara-Reid added 4 commits January 10, 2025 12:45

sonic: bump tagged version

ab5fd01

encoding/json: drop build tag arch and os filters

1697977

encoding/json: consolidate tests

3905b62

encoding/json: log build tag usage

1c9f9c2

shazbert requested a review from gloriousCode January 10, 2025 02:22

shazbert added 2 commits January 14, 2025 10:26

Merge branch 'master' into encoding_swap

4f8e487

rm superfluous builds

9cffc8f

shazbert requested a review from thrasher- January 13, 2025 23:33

shazbert mentioned this pull request Jan 14, 2025

GateIO: Fix GetFuturesContractDetails for Deliveries and minor other fixes #1766

Merged

4 tasks

gloriousCode requested changes Jan 27, 2025

View reviewed changes

shazbert added 4 commits January 28, 2025 12:14

glorious/nits: add template change and regen docs

1b15e4c

glorious/nits: update commentary on nolint directive

a7ac15d

glorious/nits: rm init func and log results in main.go

f23f3c3

Test to actually pull flag in

fba5316

shazbert requested a review from gloriousCode January 28, 2025 02:01

gloriousCode removed the review me This pull request is ready for review label Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

encoding/json: Add Custom JSON Package with Build Tag Support for Sonic #1623

encoding/json: Add Custom JSON Package with Build Tag Support for Sonic #1623

shazbert commented Aug 20, 2024

codecov bot commented Aug 20, 2024 •

edited

Loading

shazbert commented Oct 10, 2024

gloriousCode left a comment •

edited

Loading

gloriousCode Aug 20, 2024

shazbert Jan 10, 2025

thrasher- Jan 13, 2025

shazbert Jan 13, 2025

gloriousCode Jan 9, 2025

shazbert Jan 10, 2025

gloriousCode left a comment

gloriousCode Jan 27, 2025

gloriousCode Jan 27, 2025 •

edited

Loading

gloriousCode Jan 27, 2025

gloriousCode Jan 27, 2025

gloriousCode Jan 27, 2025

shazbert Jan 28, 2025

shazbert Jan 28, 2025

gloriousCode Jan 28, 2025


		package json

		import "encoding/json" //nolint:depguard // This is a wrapper package

		@@ -0,0 +1,13 @@
		package json

		import "encoding/json" //nolint:depguard // This is a wrapper package

encoding/json: Add Custom JSON Package with Build Tag Support for Sonic #1623

Are you sure you want to change the base?

encoding/json: Add Custom JSON Package with Build Tag Support for Sonic #1623

Conversation

shazbert commented Aug 20, 2024

PR Description

Type of change

How has this been tested

Checklist

codecov bot commented Aug 20, 2024 • edited Loading

Codecov Report

shazbert commented Oct 10, 2024

gloriousCode left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gloriousCode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gloriousCode Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Aug 20, 2024 •

edited

Loading

gloriousCode left a comment •

edited

Loading

gloriousCode Jan 27, 2025 •

edited

Loading